Open Access 



Research 



BMJ Open 



To cite: Panesar SS, 
Netuveli G, Carson- 
Stevens A, et al. The 
orthopaedic error index: 
development and application 
of a novel national indicator 
for assessing the relative 
safety of hospital care using 
a cross-sectional approach. 
BMJ Open 201 3;3:e003448. 
doi:1 0.1 1 36/bmjopen-201 3- 
003448 

► Prepublication history and 
additional material for this 
paper is available online. To 
view these files please visit 
the journal online 
(http://dx.doi.org/10.1136/ 
bmjopen-201 3-003448). 



Received 20 June 2013 
Accepted 11 October 2013 



CrossMark 



For numbered affiliations see 
end of article. 



Correspondence to 

Dr Sukhmeet S Panesar; 
sukhmeet. panesar® 
surgicalmatrix.com 




The orthopaedic error index: 
development and application of a novel 
national indicator for assessing the 
relative safety of hospital care using 
a cross-sectional approach 

Sukhmeet S Panesar, 1,2 Gopalakrishnan Netuveli, 3 Andrew Carson-Stevens, 4 
Sundas Javad, 5 Bhavesh Patel, 6 Gareth Parry, 7,8 Liam J Donaldson, 9 
Aziz Sheikh 2 



ABSTRACT 

Objective: The Orthopaedic Error Index for hospitals 
aims to provide the first national assessment of the 
relative safety of provision of orthopaedic surgery. 
Design: Cross-sectional study (retrospective analysis 
of records in a database). 

Setting: The National Reporting and Learning System 
is the largest national repository of patient-safety 
incidents in the world with over eight million error 
reports. It offers a unique opportunity to develop novel 
approaches to enhancing patient safety, including 
investigating the relative safety of different healthcare 
providers and specialties. 
Participants: We extracted all orthopaedic error 
reports from the system over 1 year (2009-2010). 
Outcome measures: The Orthopaedic Error Index 
was calculated as a sum of the error propensity and 
severity. All relevant hospitals offering orthopaedic 
surgery in England were then ranked by this 
metric to identify possible outliers that warrant further 
attention. 

Results: 155 hospitals reported 48 971 orthopaedic- 
related patient-safety incidents. The mean Orthopaedic 
Error Index was 7.09/year (SD 2.72); five hospitals 
were identified as outliers. Three of these units were 
specialist tertiary hospitals carrying out complex 
surgery; the remaining two outlier hospitals had 
unusually high Orthopaedic Error Indexes: mean 14.46 
(SD 0.29) and 15.29 (SD 0.51), respectively. 
Conclusions: The Orthopaedic Error Index has 
enabled identification of hospitals that may be putting 
patients at disproportionate risk of orthopaedic-related 
iatrogenic harm and which therefore warrant further 
investigation. It provides the prototype of a summary 
index of harm to enable surveillance of unsafe care 
over time across institutions. Further validation and 
scrutiny of the method will be required to assess its 
potential to be extended to other hospital specialties in 
the UK and also internationally to other health systems 
that have comparable national databases of patient- 
safety incidents. 



Strengths and limitations of this study 



■ Several notable strengths of the study: a large 
national dataset was drawn upon; comprising of 
over 48 000 orthopaedic reports and the data 
used are specifically for patient safety. 

■ There is a trade-off between the depth and 
breadth of the analysis. 

■ Existing learning from national patient-safety 
reporting systems is limited; some of the infor- 
mation is lost in translation and it is unclear 
whether all patient-safety incidents are indeed 
reported. 

■ Secondary analysis of data, including the 
absence of specific information needed and 
necessities of using proxies. 

■ Biases exist at several levels: reporting of harmful 
versus non-harmful events and correct classifica- 
tion of categories of harm. Underlying factors for 
these biases, such as level of patient-safety culture 
within institutions, were not assessed. 

INTRODUCTION 

The delivery of safer healthcare remains a 
challenge globally. 1 Over a decade ago, the 
Institute of Medicine published the seminal 
report, 'To err is human', 1 which revealed the 
previously under-recognised high burden of 
morbidity and mortality associated with iatro- 
genic harm. This was then followed by the 
equally influential Crossing the Quality Chasm, 
which highlighted the need to develop and 
make greater use of error reporting systems to 
enable the generation of learning from 
patient-safety incidents and inform opportun- 
ities for system-level interventions to reduce 
future risks of harm. 2 The challenge for 
healthcare systems globally remains the consist- 
ent delivery of safer care and the associated 
surveillance of safety within an organisation. 
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Modern day healthcare involves an array of compli- 
cated diagnostic and therapeutic decisions affected by 
the system within which these occur. Poorly functioning 
systems and teams have the potential to impact on 
patient safety. Inevitably, there will be unexpected vari- 
ation in access, outcomes and quality of care. Whereas 
some variation is legitimate and indeed desirable (eg, 
slower surgeons should not be asked to work faster) , it is 
the unwarranted variation that is a major cause of 
concern to policy-makers and regulators. 3 A much- 
favoured approach for describing this variability is the 
use of hospital-wide mortality rates. Proponents have 
argued that these tools provide useful metrics about pro- 
blems with the quality of inpatient care; uncover system- 
wide failures and can help patients to choose the safest 
hospital. 4 

Patient-safety reporting systems are unused sources of 
data for the surveillance of harm which enable learning 
from errors, so that the insights create opportunities for 
system-level interventions to reduce future risks of harm. 
Some successes have been reported, for example, in 
identifying previously undetected risks of new drugs or 
procedures, but there remain doubts about the wider 
value of investments in developing and maintaining 
large-scale incident reporting systems. 5 Databases of 
error reports now exist in many parts of the world, 
including the UK and the USA. 6-8 The UK's National 
Reporting and Learning System (NRLS) was launched 
in 2003 and has since accrued over eight million 
records, making it the most substantial repository of 
patient-safety incidents in the world. Analyses of this 
database have revealed risks in a range of clinical 
areas, 9 10 but what has been lacking are high-level, valid 
summary metrics to allow surveillance of harm across a 
whole healthcare system in a way that allows, for 
example: comparison between hospitals; monitoring of 
time trends and a baseline for assessing and evaluating 
interventions. Some doubts exist about the wider value 
of investments in developing and maintaining large-scale 
incident reporting systems. 5 

In this paper, we report on the development of one 
such summary statistic, which aims to (unlike the cur- 
rently used proxy measures of harm such as Hospital 
Standardised Mortality Ratios) provide a more direct 
measure of safety. 11 We developed and tested this 
measure in the field of orthopaedic surgery. This spe- 
cialty was chosen because it is associated with a relatively 
high level of harm. 12 For example, from 2000 to 2006, 
an equivalent of US$321 million was paid in adult ortho- 
paedic surgery-related negligence settlements in the 
UK. 13 Similar figures were reported the Physician 
Insurers Association of America. 14 



METHODS 

Developing a model for an error index 

Errors will occur in complex systems such as healthcare, 
and their frequency will relate to the number of 



procedures undertaken. Such assumptions are made in 
other fields of risk. In road traffic accidents, for 
example, predicted crash frequency is a linear function 
of average daily traffic. 15 

A simple measure of error is the frequency of errors 
occurring in any hospital. However, since frequency is a 
function of the number of procedures that have been 
carried out, we therefore deemed the frequency of 
errors per unit of procedure as a more appropriate 
measure. 16 We call this the error propensity. In order to 
calculate this, we extracted data on all orthopaedic 
reports made by all English hospitals reporting to the 
UK's NRLS over a 12-month period from 1 April 2009 to 
31 March 2010. In parallel, we approximated the total 
number of orthopaedic procedures carried out in each 
hospital using data from the national Hospital Episode 
Statistics (HES; 2009-2010) database, 17 which is a man- 
datory national database of all patient visits to National 
Health Service (NHS) hospitals in England, in order to 
estimate the error propensity. We defined an ortho- 
paedic procedure as any patient entry that involves an 
Office of Population Censuses and Surveys Classification 
of Interventions and Procedures code in the OP1 field 
of HES with a treatment specialty code of 110. 

A second component of the Orthopaedic Error Index 
(OEI) reflects the impact the error has on the patients, 
that is, how much harm it caused. We call this the error 
severity, which is based on categories of harm. Harm was 
defined by user self-reports; the degree of harm was clas- 
sified as either: no harm, low harm (minimal harm — 
patient (s) required extra observation or minor treat- 
ment), moderate harm (short-term harm — patient (s) 
required further treatment, or procedure), severe harm 
(permanent or long-term harm) or death. Further 
details about the structure of the NRLS are provided in 
online supplementary appendix 1. We created our 
summary statistic using principles laid out in the 
Standards for Statistical Models used for Public 
Reporting of Health Outcomes. 18 

The OEI comprised the two main domains of error: 
the propensity of errors (P) and the severity of harm 
(S). The mathematical derivation of the OEI is given in 
online supplementary appendix 2. 

Analyses 

We estimated P, S and OEI and their SEs for each hos- 
pital using STATAV.ll (StataCorp 2011; Stata Statistical 
Software: Release 12, College Station, Texas, USA: 
StataCorp LP). Since P, S and the OEI deviated from 
normality, they were first transformed: OEI using a loga- 
rithmic transformation and propensity and severity by 
taking their reciprocal values. We sought to identify out- 
liers by creating control lines at one, two and three SDs. 
We plotted OEI (per 1000 procedures) against number 
of procedures and superimposed lines representing the 
mean and ±2SD and 3SD of predicted OEI values. 
Funnel plots provided a visual representation of the data 
and have been used widely in health services research to 
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compare institutions. We defined outliers as those with 
an OEI outside the range of u±3g, where u is the mean 
and o is the SD for the whole sample. These hospitals 
require closer scrutiny. 19 

RESULTS 

OEI for all hospitals in England 

In England, 155 NHS hospitals reported 48 971 patient- 
safety incidents with varying degrees of harm: 34 530/ 
48 971 (70.5%) no harm; 11 529/48 971 (23.5%) low 
harm; 2632/48 971 (5.4%) moderate harm; 217/48 971 
(0.4%) severe harm and 63/48 971 (0.1%) deaths in the 
specialty of trauma and orthopaedic surgery during 
2009-2010. The mean hospital OEI was 7.09 (SD 2.72). 
There was a correlation between the number of proce- 
dures and error reports of 0.40; therefore an increase of 
1000 orthopaedic procedures generated approximately 
38 additional error reports ( figure 1 ) . 

Identifying outlier hospitals 

Among the 155 hospitals, 5 lay outside the prespecified 
control limits (figure 2). These were hospitals that had 
relatively small numbers of procedures, but high OEI 
values. Of note, there is an almost linear association with 
larger hospitals having fewer errors. 

DISCUSSION 

The OEI is the first attempt to develop automated proce- 
dures to interrogate a national database of patient-safety 
incidents in order to identify hospitals at disproportionate 
risk of iatrogenic harm. Applying this tool to all hospitals 
providing orthopaedic care identified five outlying hospi- 
tals: three tertiary care providers and two secondary care 
providers. While the higher rates may be expected 
because of case-mix considerations in the tertiary care 
sites, such deviations are not to be expected in the sec- 
ondary care providers. 

One of these two secondary care providers has subse- 
quently been highlighted nationally as providing sub- 
standard care due to failures in administration, 
management and nursing. 20 

At present, the NHS and other health systems inter- 
nationally lack direct indicators of safety. Mortality is a 
proxy measure and cannot be used in isolation to assess 
the safety of a hospital. Opponents argue that the con- 
struction of Hospital Standardised Mortality Ratios is 
flawed as the index is unable to discriminate between 
inevitable and preventable deaths, and that the huge 
variability in care suggested by these metrics cannot be 
accounted for by variable quality of care alone. 11 
Although we did not distinguish between avoidable and 
non-avoidable incidents in our analysis, previous work by 
the group has shown that most orthopaedic incidents 
that result in harm could have been prevented if safety 
measures had been implemented. 21 

The process of identifying outliers could be associated 
with stigma and extensive resource allocation, both 



financial and reputational. Nevertheless, in organisa- 
tions that foster a culture of reporting and learning, this 
method should be viewed as one of the many tools that 
need to be used in parallel to understanding how unsafe 
the care is in that particular organisation. The use of 
outlier analysis in singularity does not ensure safe care; 
it merely acts as a trigger for further checks. One of the 
hospitals that we identified as an outlier, and that had 
been the subject of national inquiries, was also noted to 
have a high Hospital Standardised Mortality Ratio; an 
excess of up to 1200 deaths occurred here. 22 We have 
thus shown that a patient-safety reporting system, which 
until recently has been used as a repository collecting 
reports of errors, can be used to identify institutions that 
may pose a disproportionate risk to patient safety. 
However, the work requires greater scrutiny and valid- 
ation; the purpose of this undertaking was to see if 
national patient-safety reporting systems can be used for 
surveillance of unsafe care. As such, we have created a 
novel metric - the OEI - which is a direct marker of 
patient safety in individual hospitals. The thrust behind 
this idea has been the occurrence of several high-profile 
cases of hospitals such as Alder Hey, Mid-Staffordshire 
and Stockport NHS hospitals, where a catalogue of 
medical errors occurred that resulted in varying degrees 
of harm to the patients. 23 Most people would agree that 
in an era of large datasets, regulators and advisory 
bodies should have mechanisms to identify hospitals that 
are struggling to deliver high-quality care at an earlier 
stage so that corrective responses can be initiated. 
Monitoring trends in unsafe care over time would be 
invaluable. They would help, in addition to identifying 
outliers, in evaluating the effect of safety initiatives and 
case-mix of patients during different periods of the year. 
Despite alarming cases of unsafe care, it appears that 
the NHS is still ill-equipped to identify high-risk hospi- 
tals through early warning systems. 24 More recently, 
attempts have been made in the UK to identify failing 
hospitals by using nationwide surveillance tools which 
collect data prospectively: the NHS Safety Thermometer 
that collects data on four domains — venous thrombo- 
embolism, urinary tract infections, pressure ulcers and 
falls; 25 the National Surgical Quality Improvement 
Program which collects measures of outcomes to 
improve surgical care; 26 and the Global Trigger Tool 
which measures adverse events. 27 However, not all hospi- 
tals use these tools. Ours is the first tool that uses data 
from an entire national healthcare system. 

Strengths and limitations 

Errors can be caused by active failures, for example, mis- 
takes and latent conditions, such as failure of system pro- 
cesses. 28 Usually primary data from small, in-depth, 
qualitative inquiries are used to identify factors that con- 
tributed to the errors. The main strength of this 
approach is that data are specific for patient safety, but a 
major limitation is the trade-off between the depth and 
breadth of the analysis. We sought to investigate the use 
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Figure 1 Relationship between 
number of error reports and 
volume of procedures. 
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of routinely collected patient-safety incident reports to 
create a numerical index that could help to provide a 
complementary perspective by supporting the monitor- 
ing of the overall system-wide safety of healthcare provi- 
sion. Other key strengths of our work include drawing 
on a large national dataset, comprising of over 48 000 
orthopaedic reports, and utilising the full body of data 
reporting on patient-safety incidents spanning the full 
spectrum of severity from no harm to death. At present, 
the learning from national patient-safety reporting 
systems is limited; some of the information is lost in 
translation and it is unclear whether all patient-safety 
incidents are indeed reported. 29 The sensitivity of the 
NRLS at picking up errors has been questioned in the 
past 30 the low power of the study limits generalisability 
of results to the entire NRLS. Furthermore, all hospitals 
use the same mechanism of reporting incidents, so the 
effects of bias and uncertainty are limited with the OEI. 
Nevertheless this should not deter exploratory work 
such as ours. We are also cognisant of the fact that there 
is likely to be a variation in reporting according to the 



patient-safety culture within hospitals; so a hospital with 
a high OEI may be one that has an open culture and 
encourages staff to report patient-safety incidents. 31 Of 
equal importance is the fact that NRLS was a voluntary 
reporting system until April 2010, when mandatory 
reporting was introduced for serious untoward inci- 
dents. 32 In figure 2, we showed that large hospitals 
(number of orthopaedic procedures) are associated with 
fewer errors. This must be interpreted with caution as 
we have not been able to adjust for patient or procedure 
case-mix due to the paucity and anonymity of the data. 
Based on work elsewhere, it has been stipulated that spe- 
cialised surgical services should be provided in tertiary 
hospitals, although geographical or logistical impedi- 
ments may occur. 33 We cannot make this claim based on 
our findings. Some local systems of risk management in 
hospitals opt for root cause analyses to develop local 
solutions to mitigate harm to patients, but these are not 
shared nationally, and limited information may be pro- 
vided to national reporting systems. These systems rely 
on patient-safety experts methodically trawling through 
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Figure 2 Orthopaedic Error Index for all hospitals in England. 
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patient-safety incidents by severity and frequency 
thereby leading to the production of quarterly reports, 
alerts and rapid response solutions. 34 Such analyses are 
time-consuming and, as the number of reports rapidly 
increases, may in the future be unsustainable. 35 36 There 
is also a non-cohesive approach globally to identifying 
unsafe hospitals. The multitude of quality indicators that 
are proxy measures of unsafe care is overwhelming. The 
OEI is a surveillance tool that can enable direct evalu- 
ation of safer care in hospitals. It is, we believe, a novel 
benchmarking tool to assess patient safety across hospi- 
tals using a large patient-safety reporting system. 

The main limitations are those inherent to any sec- 
ondary analysis of data, including the absence of specific 
information needed and necessities of using proxies. 
Ideally, we would have preferred to link HES data to cor- 
responding NRLS incidents. At present, this is not pos- 
sible, as the latter does not allow for patient 
identification via NHS identification numbers. HES data 
will also give an approximation of orthopaedic and 
trauma procedures due to coding inaccuracies. However, 
these are, we believe, largely mitigated in the present 
analysis by the fact that the data were collected to study 
error and we refer to our analyses as secondary only 
because the analysis approach we employed was 
unanticipated when the study was designed. However, 
the OEI has several potential limitations. Reporting of 
patient-safety incidents is a subjective exercise and vari- 
ation in the dataset is bound to exist. Biases also exist at 
several levels: reporting of harmful versus non-harmful 
events and correct classification of categories of harm. 37 
Underlying factors for these biases, such as the level of 
patient-safety culture within institutions, were not 
assessed. 38 Further work on measuring the extent and 
likely impact of such biases is therefore now needed. 

CONCLUSIONS 

With the proliferation of patient-safety reporting systems 
around the world and an ever-increasing number of 
patient-safety incidents reported to them, sophisticated 
analytical techniques are required to identify hospitals 
that need to strengthen their emphasis on patient safety. 
This is the first time, to our knowledge, that a surveil- 
lance mechanism for safety has been proposed using a 
reporting system. 
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